Empirical Evaluation of the Communicative Effectiveness of an Automatic Speech-to-Speech Translation System

In this study we evaluate the ability of a state-of-the-art speechto-speech machine translation system, Skype Translator, to preserve the communicative effectiveness of its input speech. Specifically, we elicited Spanish spoken responses from 24 native speakers of Spanish in the following two scenarios: academic speaking (in the context of 6 questions from a standardized assessment of academic speaking proficiency) and daily communication (in the context of 6 map-based directions tasks). Skype Translator was then used to translate these Spanish responses into English, and expert human raters subsequently provided speaking proficiency scores using both holistic and analytic scoring rubrics with a 1-4 range for both the original Spanish responses and the corresponding automatically translated English versions. The results show that the average holistic scores for the academic speaking task decreased from 3.5 (for the original Spanish responses) to 1.9 (for the English translations) on a 4-point scale, and the average task completion scores for the map-based tasks decreased from 3.4 to 2.0. While these results show that automated speech-to-speech machine translation does not yet fully meet the communicative demands of these speaking tasks, they also indicate how far the technology has improved in recent years and can serve as a benchmark for tracking further improvements in the future.